Towards Multi-Modal Interactions in Virtual Environments: A Case Study

We discuss interaction modalities in web-based virtual environments. We argue that the language behaviour of users in virtual environments differs essentially from language behaviour in the context of other spoken dialogue systems. Allowing different modalities will also ask for a model that allows user and system to expect that knowledge obtained from previous interactions is available to whatever ‘agents’ are addressed by the user. We survey our research on interaction modalities in a theatre information and booking system. A first version of the system allowed natural language interaction using keyboard only. Current research has made it possible to allow other modalities. Rather than presenting questions and answers on the screen, we allow output using speech synthesis in combination with screen-based information. Since the system has been embedded in a virtual reality environment, which models a theatre, it becomes possible to investigate interactions between users and theatre agents.